Second-Order Step-Size Tuning of SGD for Non-Convex Optimization

In view of a direct and simple improvement vanilla SGD, this paper presents fine-tuning its step-sizes in the mini-batch case. For doing so, one estimates curvature, based on local quadratic model using only noisy gradient approximations. One obtains new stochastic first-order method (Step-Tuned SGD), enhanced by second-order information, which can be seen as version classical Barzilai-Borwein method. Our theoretical results ensure almost sure convergence to critical set we provide rates. Experiments deep residual network training illustrate favorable properties our approach. such networks observe, during training, both sudden drop loss an test accuracy at medium stages, yielding better than RMSprop, or ADAM.